{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6df76a1f",
   "metadata": {},
   "source": [
    "# Using MLflow with Tune\n",
    "\n",
    "<a id=\"try-anyscale-quickstart-tune-mlflow\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-mlflow\">\n",
    "    <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
    "</a>\n",
    "<br></br>\n",
    "\n",
    "(tune-mlflow-ref)=\n",
    "\n",
    "[MLflow](https://mlflow.org/) is an open source platform to manage the ML lifecycle, including experimentation,\n",
    "reproducibility, deployment, and a central model registry. It currently offers four components, including\n",
    "MLflow Tracking to record and query experiments, including code, data, config, and results.\n",
    "\n",
    "```{image} /images/mlflow.png\n",
    ":align: center\n",
    ":alt: MLflow\n",
    ":height: 80px\n",
    ":target: https://www.mlflow.org/\n",
    "```\n",
    "\n",
    "Ray Tune currently offers two lightweight integrations for MLflow Tracking.\n",
    "One is the {ref}`MLflowLoggerCallback <tune-mlflow-logger>`, which automatically logs\n",
    "metrics reported to Tune to the MLflow Tracking API.\n",
    "\n",
    "The other one is the {ref}`setup_mlflow <tune-mlflow-setup>` function, which can be\n",
    "used with the function API. It automatically\n",
    "initializes the MLflow API with Tune's training information and creates a run for each Tune trial.\n",
    "Then within your training function, you can just use the\n",
    "MLflow like you would normally do, e.g. using `mlflow.log_metrics()` or even `mlflow.autolog()`\n",
    "to log to your training process.\n",
    "\n",
    "```{contents}\n",
    ":backlinks: none\n",
    ":local: true\n",
    "```\n",
    "\n",
    "## Running an MLflow Example\n",
    "\n",
    "In the following example we're going to use both of the above methods, namely the `MLflowLoggerCallback` and\n",
    "the `setup_mlflow` function to log metrics.\n",
    "Let's start with a few crucial imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b0e47339",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import tempfile\n",
    "import time\n",
    "\n",
    "import mlflow\n",
    "\n",
    "from ray import tune\n",
    "from ray.air.integrations.mlflow import MLflowLoggerCallback, setup_mlflow\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "618b6935",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Next, let's define an easy training function (a Tune `Trainable`) that iteratively computes steps and evaluates\n",
    "intermediate scores that we report to Tune."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f449538e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def evaluation_fn(step, width, height):\n",
    "    return (0.1 + width * step / 100) ** (-1) + height * 0.1\n",
    "\n",
    "\n",
    "def train_function(config):\n",
    "    width, height = config[\"width\"], config[\"height\"]\n",
    "\n",
    "    for step in range(config.get(\"steps\", 100)):\n",
    "        # Iterative training function - can be any arbitrary training procedure\n",
    "        intermediate_score = evaluation_fn(step, width, height)\n",
    "        # Feed the score back to Tune.\n",
    "        tune.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n",
    "        time.sleep(0.1)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "722e5d2f",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Given an MLFlow tracking URI, you can now simply use the `MLflowLoggerCallback` as a `callback` argument to\n",
    "your `RunConfig()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8e0b9ab7",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def tune_with_callback(mlflow_tracking_uri, finish_fast=False):\n",
    "    tuner = tune.Tuner(\n",
    "        train_function,\n",
    "        tune_config=tune.TuneConfig(num_samples=5),\n",
    "        run_config=tune.RunConfig(\n",
    "            name=\"mlflow\",\n",
    "            callbacks=[\n",
    "                MLflowLoggerCallback(\n",
    "                    tracking_uri=mlflow_tracking_uri,\n",
    "                    experiment_name=\"mlflow_callback_example\",\n",
    "                    save_artifact=True,\n",
    "                )\n",
    "            ],\n",
    "        ),\n",
    "        param_space={\n",
    "            \"width\": tune.randint(10, 100),\n",
    "            \"height\": tune.randint(0, 100),\n",
    "            \"steps\": 5 if finish_fast else 100,\n",
    "        },\n",
    "    )\n",
    "    results = tuner.fit()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e086f110",
   "metadata": {},
   "source": [
    "To use the `setup_mlflow` utility, you simply call this function in your training function.\n",
    "Note that we also use `mlflow.log_metrics(...)` to log metrics to MLflow.\n",
    "Otherwise, this version of our training function is identical to its original."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "144b8f39",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def train_function_mlflow(config):\n",
    "    tracking_uri = config.pop(\"tracking_uri\", None)\n",
    "    setup_mlflow(\n",
    "        config,\n",
    "        experiment_name=\"setup_mlflow_example\",\n",
    "        tracking_uri=tracking_uri,\n",
    "    )\n",
    "\n",
    "    # Hyperparameters\n",
    "    width, height = config[\"width\"], config[\"height\"]\n",
    "\n",
    "    for step in range(config.get(\"steps\", 100)):\n",
    "        # Iterative training function - can be any arbitrary training procedure\n",
    "        intermediate_score = evaluation_fn(step, width, height)\n",
    "        # Log the metrics to mlflow\n",
    "        mlflow.log_metrics(dict(mean_loss=intermediate_score), step=step)\n",
    "        # Feed the score back to Tune.\n",
    "        tune.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n",
    "        time.sleep(0.1)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dc480366",
   "metadata": {},
   "source": [
    "With this new objective function ready, you can now create a Tune run with it as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4b9fe6be",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def tune_with_setup(mlflow_tracking_uri, finish_fast=False):\n",
    "    # Set the experiment, or create a new one if does not exist yet.\n",
    "    mlflow.set_tracking_uri(mlflow_tracking_uri)\n",
    "    mlflow.set_experiment(experiment_name=\"setup_mlflow_example\")\n",
    "\n",
    "    tuner = tune.Tuner(\n",
    "        train_function_mlflow,\n",
    "        tune_config=tune.TuneConfig(num_samples=5),\n",
    "        run_config=tune.RunConfig(\n",
    "            name=\"mlflow\",\n",
    "        ),\n",
    "        param_space={\n",
    "            \"width\": tune.randint(10, 100),\n",
    "            \"height\": tune.randint(0, 100),\n",
    "            \"steps\": 5 if finish_fast else 100,\n",
    "            \"tracking_uri\": mlflow.get_tracking_uri(),\n",
    "        },\n",
    "    )\n",
    "    results = tuner.fit()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "915dfd30",
   "metadata": {},
   "source": [
    "If you hapen to have an MLFlow tracking URI, you can set it below in the `mlflow_tracking_uri` variable and set\n",
    "`smoke_test=False`.\n",
    "Otherwise, you can just run a quick test of the `tune_function` and `tune_decorated` functions without using MLflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "05d11774",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-12-22 10:37:53,580\tINFO worker.py:1542 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div class=\"tuneStatus\">\n",
       "  <div style=\"display: flex;flex-direction: row\">\n",
       "    <div style=\"display: flex;flex-direction: column;\">\n",
       "      <h3>Tune Status</h3>\n",
       "      <table>\n",
       "<tbody>\n",
       "<tr><td>Current time:</td><td>2022-12-22 10:38:04</td></tr>\n",
       "<tr><td>Running for: </td><td>00:00:06.73        </td></tr>\n",
       "<tr><td>Memory:      </td><td>10.4/16.0 GiB      </td></tr>\n",
       "</tbody>\n",
       "</table>\n",
       "    </div>\n",
       "    <div class=\"vDivider\"></div>\n",
       "    <div class=\"systemInfo\">\n",
       "      <h3>System Info</h3>\n",
       "      Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.03 GiB heap, 0.0/2.0 GiB objects\n",
       "    </div>\n",
       "    \n",
       "  </div>\n",
       "  <div class=\"hDivider\"></div>\n",
       "  <div class=\"trialStatus\">\n",
       "    <h3>Trial Status</h3>\n",
       "    <table>\n",
       "<thead>\n",
       "<tr><th>Trial name                </th><th>status    </th><th>loc          </th><th style=\"text-align: right;\">  height</th><th style=\"text-align: right;\">  width</th><th style=\"text-align: right;\">   loss</th><th style=\"text-align: right;\">  iter</th><th style=\"text-align: right;\">  total time (s)</th><th style=\"text-align: right;\">  iterations</th><th style=\"text-align: right;\">  neg_mean_loss</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>train_function_b275b_00000</td><td>TERMINATED</td><td>127.0.0.1:801</td><td style=\"text-align: right;\">      66</td><td style=\"text-align: right;\">     36</td><td style=\"text-align: right;\">7.24935</td><td style=\"text-align: right;\">     5</td><td style=\"text-align: right;\">        0.587302</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">       -7.24935</td></tr>\n",
       "<tr><td>train_function_b275b_00001</td><td>TERMINATED</td><td>127.0.0.1:813</td><td style=\"text-align: right;\">      33</td><td style=\"text-align: right;\">     35</td><td style=\"text-align: right;\">3.96667</td><td style=\"text-align: right;\">     5</td><td style=\"text-align: right;\">        0.507423</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">       -3.96667</td></tr>\n",
       "<tr><td>train_function_b275b_00002</td><td>TERMINATED</td><td>127.0.0.1:814</td><td style=\"text-align: right;\">      75</td><td style=\"text-align: right;\">     29</td><td style=\"text-align: right;\">8.29365</td><td style=\"text-align: right;\">     5</td><td style=\"text-align: right;\">        0.518995</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">       -8.29365</td></tr>\n",
       "<tr><td>train_function_b275b_00003</td><td>TERMINATED</td><td>127.0.0.1:815</td><td style=\"text-align: right;\">      28</td><td style=\"text-align: right;\">     63</td><td style=\"text-align: right;\">3.18168</td><td style=\"text-align: right;\">     5</td><td style=\"text-align: right;\">        0.567739</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">       -3.18168</td></tr>\n",
       "<tr><td>train_function_b275b_00004</td><td>TERMINATED</td><td>127.0.0.1:816</td><td style=\"text-align: right;\">      20</td><td style=\"text-align: right;\">     18</td><td style=\"text-align: right;\">3.21951</td><td style=\"text-align: right;\">     5</td><td style=\"text-align: right;\">        0.526536</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">       -3.21951</td></tr>\n",
       "</tbody>\n",
       "</table>\n",
       "  </div>\n",
       "</div>\n",
       "<style>\n",
       ".tuneStatus {\n",
       "  color: var(--jp-ui-font-color1);\n",
       "}\n",
       ".tuneStatus .systemInfo {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       ".tuneStatus td {\n",
       "  white-space: nowrap;\n",
       "}\n",
       ".tuneStatus .trialStatus {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       ".tuneStatus h3 {\n",
       "  font-weight: bold;\n",
       "}\n",
       ".tuneStatus .hDivider {\n",
       "  border-bottom-width: var(--jp-border-width);\n",
       "  border-bottom-color: var(--jp-border-color0);\n",
       "  border-bottom-style: solid;\n",
       "}\n",
       ".tuneStatus .vDivider {\n",
       "  border-left-width: var(--jp-border-width);\n",
       "  border-left-color: var(--jp-border-color0);\n",
       "  border-left-style: solid;\n",
       "  margin: 0.5em 1em 0.5em 1em;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div class=\"trialProgress\">\n",
       "  <h3>Trial Progress</h3>\n",
       "  <table>\n",
       "<thead>\n",
       "<tr><th>Trial name                </th><th>date               </th><th>done  </th><th>episodes_total  </th><th>experiment_id                   </th><th>experiment_tag      </th><th>hostname                                        </th><th style=\"text-align: right;\">  iterations</th><th style=\"text-align: right;\">  iterations_since_restore</th><th style=\"text-align: right;\">  mean_loss</th><th style=\"text-align: right;\">  neg_mean_loss</th><th>node_ip  </th><th style=\"text-align: right;\">  pid</th><th style=\"text-align: right;\">  time_since_restore</th><th style=\"text-align: right;\">  time_this_iter_s</th><th style=\"text-align: right;\">  time_total_s</th><th style=\"text-align: right;\">  timestamp</th><th style=\"text-align: right;\">  timesteps_since_restore</th><th>timesteps_total  </th><th style=\"text-align: right;\">  training_iteration</th><th>trial_id   </th><th style=\"text-align: right;\">  warmup_time</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>train_function_b275b_00000</td><td>2022-12-22_10-38-01</td><td>True  </td><td>                </td><td>28feaa4dd8ab4edab810e8109e77502e</td><td>0_height=66,width=36</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">                         5</td><td style=\"text-align: right;\">    7.24935</td><td style=\"text-align: right;\">       -7.24935</td><td>127.0.0.1</td><td style=\"text-align: right;\">  801</td><td style=\"text-align: right;\">            0.587302</td><td style=\"text-align: right;\">          0.126818</td><td style=\"text-align: right;\">      0.587302</td><td style=\"text-align: right;\"> 1671705481</td><td style=\"text-align: right;\">                        0</td><td>                 </td><td style=\"text-align: right;\">                   5</td><td>b275b_00000</td><td style=\"text-align: right;\">   0.00293493</td></tr>\n",
       "<tr><td>train_function_b275b_00001</td><td>2022-12-22_10-38-04</td><td>True  </td><td>                </td><td>245010d0c3d0439ebfb664764ae9db3c</td><td>1_height=33,width=35</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">                         5</td><td style=\"text-align: right;\">    3.96667</td><td style=\"text-align: right;\">       -3.96667</td><td>127.0.0.1</td><td style=\"text-align: right;\">  813</td><td style=\"text-align: right;\">            0.507423</td><td style=\"text-align: right;\">          0.122086</td><td style=\"text-align: right;\">      0.507423</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\">                        0</td><td>                 </td><td style=\"text-align: right;\">                   5</td><td>b275b_00001</td><td style=\"text-align: right;\">   0.00553799</td></tr>\n",
       "<tr><td>train_function_b275b_00002</td><td>2022-12-22_10-38-04</td><td>True  </td><td>                </td><td>898afbf9b906448c980f399c72a2324c</td><td>2_height=75,width=29</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">                         5</td><td style=\"text-align: right;\">    8.29365</td><td style=\"text-align: right;\">       -8.29365</td><td>127.0.0.1</td><td style=\"text-align: right;\">  814</td><td style=\"text-align: right;\">            0.518995</td><td style=\"text-align: right;\">          0.123554</td><td style=\"text-align: right;\">      0.518995</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\">                        0</td><td>                 </td><td style=\"text-align: right;\">                   5</td><td>b275b_00002</td><td style=\"text-align: right;\">   0.0040431 </td></tr>\n",
       "<tr><td>train_function_b275b_00003</td><td>2022-12-22_10-38-04</td><td>True  </td><td>                </td><td>03a4476f82734642b6ab0a5040ca58f8</td><td>3_height=28,width=63</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">                         5</td><td style=\"text-align: right;\">    3.18168</td><td style=\"text-align: right;\">       -3.18168</td><td>127.0.0.1</td><td style=\"text-align: right;\">  815</td><td style=\"text-align: right;\">            0.567739</td><td style=\"text-align: right;\">          0.125471</td><td style=\"text-align: right;\">      0.567739</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\">                        0</td><td>                 </td><td style=\"text-align: right;\">                   5</td><td>b275b_00003</td><td style=\"text-align: right;\">   0.00406194</td></tr>\n",
       "<tr><td>train_function_b275b_00004</td><td>2022-12-22_10-38-04</td><td>True  </td><td>                </td><td>ff8c7c55ce6e404f9b0552c17f7a0c40</td><td>4_height=20,width=18</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">                         5</td><td style=\"text-align: right;\">    3.21951</td><td style=\"text-align: right;\">       -3.21951</td><td>127.0.0.1</td><td style=\"text-align: right;\">  816</td><td style=\"text-align: right;\">            0.526536</td><td style=\"text-align: right;\">          0.123327</td><td style=\"text-align: right;\">      0.526536</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\">                        0</td><td>                 </td><td style=\"text-align: right;\">                   5</td><td>b275b_00004</td><td style=\"text-align: right;\">   0.00332022</td></tr>\n",
       "</tbody>\n",
       "</table>\n",
       "</div>\n",
       "<style>\n",
       ".trialProgress {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  color: var(--jp-ui-font-color1);\n",
       "}\n",
       ".trialProgress h3 {\n",
       "  font-weight: bold;\n",
       "}\n",
       ".trialProgress td {\n",
       "  white-space: nowrap;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-12-22 10:38:04,477\tINFO tune.py:772 -- Total run time: 7.99 seconds (6.71 seconds for the tuning loop).\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div class=\"tuneStatus\">\n",
       "  <div style=\"display: flex;flex-direction: row\">\n",
       "    <div style=\"display: flex;flex-direction: column;\">\n",
       "      <h3>Tune Status</h3>\n",
       "      <table>\n",
       "<tbody>\n",
       "<tr><td>Current time:</td><td>2022-12-22 10:38:11</td></tr>\n",
       "<tr><td>Running for: </td><td>00:00:07.00        </td></tr>\n",
       "<tr><td>Memory:      </td><td>10.7/16.0 GiB      </td></tr>\n",
       "</tbody>\n",
       "</table>\n",
       "    </div>\n",
       "    <div class=\"vDivider\"></div>\n",
       "    <div class=\"systemInfo\">\n",
       "      <h3>System Info</h3>\n",
       "      Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.03 GiB heap, 0.0/2.0 GiB objects\n",
       "    </div>\n",
       "    \n",
       "  </div>\n",
       "  <div class=\"hDivider\"></div>\n",
       "  <div class=\"trialStatus\">\n",
       "    <h3>Trial Status</h3>\n",
       "    <table>\n",
       "<thead>\n",
       "<tr><th>Trial name                       </th><th>status    </th><th>loc          </th><th style=\"text-align: right;\">  height</th><th style=\"text-align: right;\">  width</th><th style=\"text-align: right;\">   loss</th><th style=\"text-align: right;\">  iter</th><th style=\"text-align: right;\">  total time (s)</th><th style=\"text-align: right;\">  iterations</th><th style=\"text-align: right;\">  neg_mean_loss</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>train_function_mlflow_b73bd_00000</td><td>TERMINATED</td><td>127.0.0.1:842</td><td style=\"text-align: right;\">      37</td><td style=\"text-align: right;\">     68</td><td style=\"text-align: right;\">4.05461</td><td style=\"text-align: right;\">     5</td><td style=\"text-align: right;\">        0.750435</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">       -4.05461</td></tr>\n",
       "<tr><td>train_function_mlflow_b73bd_00001</td><td>TERMINATED</td><td>127.0.0.1:853</td><td style=\"text-align: right;\">      50</td><td style=\"text-align: right;\">     20</td><td style=\"text-align: right;\">6.11111</td><td style=\"text-align: right;\">     5</td><td style=\"text-align: right;\">        0.652748</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">       -6.11111</td></tr>\n",
       "<tr><td>train_function_mlflow_b73bd_00002</td><td>TERMINATED</td><td>127.0.0.1:854</td><td style=\"text-align: right;\">      38</td><td style=\"text-align: right;\">     83</td><td style=\"text-align: right;\">4.0924 </td><td style=\"text-align: right;\">     5</td><td style=\"text-align: right;\">        0.6513  </td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">       -4.0924 </td></tr>\n",
       "<tr><td>train_function_mlflow_b73bd_00003</td><td>TERMINATED</td><td>127.0.0.1:855</td><td style=\"text-align: right;\">      15</td><td style=\"text-align: right;\">     93</td><td style=\"text-align: right;\">1.76178</td><td style=\"text-align: right;\">     5</td><td style=\"text-align: right;\">        0.650586</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">       -1.76178</td></tr>\n",
       "<tr><td>train_function_mlflow_b73bd_00004</td><td>TERMINATED</td><td>127.0.0.1:856</td><td style=\"text-align: right;\">      75</td><td style=\"text-align: right;\">     43</td><td style=\"text-align: right;\">8.04945</td><td style=\"text-align: right;\">     5</td><td style=\"text-align: right;\">        0.656046</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">       -8.04945</td></tr>\n",
       "</tbody>\n",
       "</table>\n",
       "  </div>\n",
       "</div>\n",
       "<style>\n",
       ".tuneStatus {\n",
       "  color: var(--jp-ui-font-color1);\n",
       "}\n",
       ".tuneStatus .systemInfo {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       ".tuneStatus td {\n",
       "  white-space: nowrap;\n",
       "}\n",
       ".tuneStatus .trialStatus {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       ".tuneStatus h3 {\n",
       "  font-weight: bold;\n",
       "}\n",
       ".tuneStatus .hDivider {\n",
       "  border-bottom-width: var(--jp-border-width);\n",
       "  border-bottom-color: var(--jp-border-color0);\n",
       "  border-bottom-style: solid;\n",
       "}\n",
       ".tuneStatus .vDivider {\n",
       "  border-left-width: var(--jp-border-width);\n",
       "  border-left-color: var(--jp-border-color0);\n",
       "  border-left-style: solid;\n",
       "  margin: 0.5em 1em 0.5em 1em;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div class=\"trialProgress\">\n",
       "  <h3>Trial Progress</h3>\n",
       "  <table>\n",
       "<thead>\n",
       "<tr><th>Trial name                       </th><th>date               </th><th>done  </th><th>episodes_total  </th><th>experiment_id                   </th><th>experiment_tag      </th><th>hostname                                        </th><th style=\"text-align: right;\">  iterations</th><th style=\"text-align: right;\">  iterations_since_restore</th><th style=\"text-align: right;\">  mean_loss</th><th style=\"text-align: right;\">  neg_mean_loss</th><th>node_ip  </th><th style=\"text-align: right;\">  pid</th><th style=\"text-align: right;\">  time_since_restore</th><th style=\"text-align: right;\">  time_this_iter_s</th><th style=\"text-align: right;\">  time_total_s</th><th style=\"text-align: right;\">  timestamp</th><th style=\"text-align: right;\">  timesteps_since_restore</th><th>timesteps_total  </th><th style=\"text-align: right;\">  training_iteration</th><th>trial_id   </th><th style=\"text-align: right;\">  warmup_time</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>train_function_mlflow_b73bd_00000</td><td>2022-12-22_10-38-08</td><td>True  </td><td>                </td><td>62703cfe82e54d74972377fbb525b000</td><td>0_height=37,width=68</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">                         5</td><td style=\"text-align: right;\">    4.05461</td><td style=\"text-align: right;\">       -4.05461</td><td>127.0.0.1</td><td style=\"text-align: right;\">  842</td><td style=\"text-align: right;\">            0.750435</td><td style=\"text-align: right;\">          0.108625</td><td style=\"text-align: right;\">      0.750435</td><td style=\"text-align: right;\"> 1671705488</td><td style=\"text-align: right;\">                        0</td><td>                 </td><td style=\"text-align: right;\">                   5</td><td>b73bd_00000</td><td style=\"text-align: right;\">   0.0030272 </td></tr>\n",
       "<tr><td>train_function_mlflow_b73bd_00001</td><td>2022-12-22_10-38-11</td><td>True  </td><td>                </td><td>03ea89852115465392ed318db8021614</td><td>1_height=50,width=20</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">                         5</td><td style=\"text-align: right;\">    6.11111</td><td style=\"text-align: right;\">       -6.11111</td><td>127.0.0.1</td><td style=\"text-align: right;\">  853</td><td style=\"text-align: right;\">            0.652748</td><td style=\"text-align: right;\">          0.110796</td><td style=\"text-align: right;\">      0.652748</td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\">                        0</td><td>                 </td><td style=\"text-align: right;\">                   5</td><td>b73bd_00001</td><td style=\"text-align: right;\">   0.00303078</td></tr>\n",
       "<tr><td>train_function_mlflow_b73bd_00002</td><td>2022-12-22_10-38-11</td><td>True  </td><td>                </td><td>3731fc2966f9453ba58c650d89035ab4</td><td>2_height=38,width=83</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">                         5</td><td style=\"text-align: right;\">    4.0924 </td><td style=\"text-align: right;\">       -4.0924 </td><td>127.0.0.1</td><td style=\"text-align: right;\">  854</td><td style=\"text-align: right;\">            0.6513  </td><td style=\"text-align: right;\">          0.108578</td><td style=\"text-align: right;\">      0.6513  </td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\">                        0</td><td>                 </td><td style=\"text-align: right;\">                   5</td><td>b73bd_00002</td><td style=\"text-align: right;\">   0.00310016</td></tr>\n",
       "<tr><td>train_function_mlflow_b73bd_00003</td><td>2022-12-22_10-38-11</td><td>True  </td><td>                </td><td>fb35841742b348b9912d10203c730f1e</td><td>3_height=15,width=93</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">                         5</td><td style=\"text-align: right;\">    1.76178</td><td style=\"text-align: right;\">       -1.76178</td><td>127.0.0.1</td><td style=\"text-align: right;\">  855</td><td style=\"text-align: right;\">            0.650586</td><td style=\"text-align: right;\">          0.109097</td><td style=\"text-align: right;\">      0.650586</td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\">                        0</td><td>                 </td><td style=\"text-align: right;\">                   5</td><td>b73bd_00003</td><td style=\"text-align: right;\">   0.0576491 </td></tr>\n",
       "<tr><td>train_function_mlflow_b73bd_00004</td><td>2022-12-22_10-38-11</td><td>True  </td><td>                </td><td>6d3cbf9ecc3446369e607ff78c67bc29</td><td>4_height=75,width=43</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\">           4</td><td style=\"text-align: right;\">                         5</td><td style=\"text-align: right;\">    8.04945</td><td style=\"text-align: right;\">       -8.04945</td><td>127.0.0.1</td><td style=\"text-align: right;\">  856</td><td style=\"text-align: right;\">            0.656046</td><td style=\"text-align: right;\">          0.109869</td><td style=\"text-align: right;\">      0.656046</td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\">                        0</td><td>                 </td><td style=\"text-align: right;\">                   5</td><td>b73bd_00004</td><td style=\"text-align: right;\">   0.00265694</td></tr>\n",
       "</tbody>\n",
       "</table>\n",
       "</div>\n",
       "<style>\n",
       ".trialProgress {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  color: var(--jp-ui-font-color1);\n",
       "}\n",
       ".trialProgress h3 {\n",
       "  font-weight: bold;\n",
       "}\n",
       ".trialProgress td {\n",
       "  white-space: nowrap;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-12-22 10:38:11,514\tINFO tune.py:772 -- Total run time: 7.01 seconds (6.98 seconds for the tuning loop).\n"
     ]
    }
   ],
   "source": [
    "smoke_test = True\n",
    "\n",
    "if smoke_test:\n",
    "    mlflow_tracking_uri = os.path.join(tempfile.gettempdir(), \"mlruns\")\n",
    "else:\n",
    "    mlflow_tracking_uri = \"<MLFLOW_TRACKING_URI>\"\n",
    "\n",
    "tune_with_callback(mlflow_tracking_uri, finish_fast=smoke_test)\n",
    "if not smoke_test:\n",
    "    df = mlflow.search_runs(\n",
    "        [mlflow.get_experiment_by_name(\"mlflow_callback_example\").experiment_id]\n",
    "    )\n",
    "    print(df)\n",
    "\n",
    "tune_with_setup(mlflow_tracking_uri, finish_fast=smoke_test)\n",
    "if not smoke_test:\n",
    "    df = mlflow.search_runs(\n",
    "        [mlflow.get_experiment_by_name(\"setup_mlflow_example\").experiment_id]\n",
    "    )\n",
    "    print(df)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f0df0817",
   "metadata": {},
   "source": [
    "This completes our Tune and MLflow walk-through.\n",
    "In the following sections you can find more details on the API of the Tune-MLflow integration.\n",
    "\n",
    "## MLflow AutoLogging\n",
    "\n",
    "You can also check out {doc}`here </tune/examples/includes/mlflow_ptl_example>` for an example on how you can\n",
    "leverage MLflow auto-logging, in this case with Pytorch Lightning\n",
    "\n",
    "## MLflow Logger API\n",
    "\n",
    "(tune-mlflow-logger)=\n",
    "\n",
    "```{eval-rst}\n",
    ".. autoclass:: ray.air.integrations.mlflow.MLflowLoggerCallback\n",
    "   :noindex:\n",
    "```\n",
    "\n",
    "## MLflow setup API\n",
    "\n",
    "(tune-mlflow-setup)=\n",
    "\n",
    "```{eval-rst}\n",
    ".. autofunction:: ray.air.integrations.mlflow.setup_mlflow\n",
    "   :noindex:\n",
    "```\n",
    "\n",
    "## More MLflow Examples\n",
    "\n",
    "- {doc}`/tune/examples/includes/mlflow_ptl_example`: Example for using [MLflow](https://github.com/mlflow/mlflow/)\n",
    "  and [Pytorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) with Ray Tune."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "orphan": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
